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© High speed segmented neural network and fabrication method. 
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. © A high speed, feed forward, segmented neural 
network includes a plurality of network layers 
stacked (12) in an ascending pyramid fashion. The 
network layers are structured with a plurality of sub- 
networks, and within each subnetwork exists a plural- 
ity of nodes (22, 24, 26) structured in a fully inter- 
connected and/or partially interconnected layered 
neural network arrangement. The inputs and outputs 
of each subnetwork are one bit digital values con- 
strained to '0* or T, while any number of nodes with 
any number of layers may be modeled for each 
subnetwork. Each subnetwork is independent of all 
other subnetworks in a given network layer, and 
thus, each network layer is segmented. In hardware 
implementation, each subnetwork comprises a sim- 
ple memory device, such as a RAM or PROM look- 
up table. The speed of the neural network system is 
high and largely dictated by the access time of the 
memory devices used. A network fabrication ap- 
proach is also presented. 
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Background of the Invention 

Field of the Invention 

The invention relates in general to the field of 
computer processing systems and, in particular, to 
high speed neural network systems and fabrication 
techniques thereof. 

Background of the Invention 

Artificial neural networks are massively parallel 
neuron-type elements that are interconnected in a 
specific manner to provide, but are not limited to, 
optical character recognition, pattern recognition, 
machine learning, process control and voice rec- 
ognition. The most common structures in artificial 
neural network systems are networks of non-linear 
processing elements, where "nodes" are intercon- 
nected to a plurality of inputs through information 
processing channels or "weights." Each node can 
process multiple inputs and weights, and each 
node has one output signal. The networks can 
often have multiple layers, wherein layers subse- 
quent to the first layer receive inputs from the 
outputs of the previous layer. The last layer in the 
network generally provides the output stimulus. 

Neural networks can simulate, on a very basic 
level, the features of biological nervous systems. 
Many of the advantages of biological nervous sys- 
tems include: the ability to generalize, adapt and 
deal with a wide degree of latitude in environments, 
operate in a massively parallel form to effectively 
function at real time rates, fault tolerance or the 
ability to deal with errors internal to the network 
itself, and the ability to learn by example. Neural 
networks do require training before useful results 
can be obtained. However, in many applications, 
one time batch back-propagation training of a neu- 
ral network is sufficient. Once trained, the resultant 
"weights" are stored and retrieved for later use in a 
non-training, testing mode or "forward mode" op- 
eration. 

In most highly interconnected networks the 
number of weights increases rapidly in a non-linear 
fashion as the number of nodes and inputs in- 
creases linearly. For example, if the number of 
nodes increases linearly with the number of inputs 
within a single layer of a fully interconnected net- 
work, then the number of weights increases as the 
square of the number of inputs. More specifically, 
in a small network layer of say 10 inputs and 10 
nodes which are fully interconnected, 100 weights 
would be employed. However, for 1000 inputs and 
1000 nodes, the number of necessary weights 
would be prohibitively high at 1,000,000. This 
would not only require massive amounts of hard- 
ware to simulate but could be very complex and 



slow in the forward mode when emulated on a 
single processor system. 

Problem to be Solved by the Invention 

5 

Tantamount to real time network operation in 
the forward mode (for example, in a pattern rec- 
ognition system using a neural network) is the use 
of very fast simplified hardware. This is often dif- 

10 ficult to achieve since many pattern recognition 
applications require a large number of inputs 
and/or nodes. Many previously developed pattern 
recognition systems do not use a simplified and 
fast hardware system. Further, in the case of an 

75 extremely large number of inputs, a true parallel 
hardware manifestation of a neural network is virtu- 
ally impossible. In many real time pattern recogni- 
tion applications, speeds in excess of many billion 
interconnects per second are required for the neu- 

20 ral network. However, a neural network that op- 
erates at these effective speeds is not currently 
available. In addition, many previously developed 
pattern recognition neural network systems have 
structures which do not lend themselves well to 

25 high speed pattern recognition in hardware im- 
plementation. 

Thus, a need exists in the art for a neural 
network system which lends itself to implementa- 
tion with a large number of inputs and nodes and 

30 which does not require an enormous number of 
weights and physical interconnections. In addition, 
the topology of the network must be such that not 
only can it be manifested in a realizable and finite 
amount of hardware, but it must also be capable of 

35 operation at very high speeds. The neural network 
system disclosed herein satisfies these require- 
ments. 

Summary of the Invention 

40 

Briefly summarized, in a first broad aspect, the 
present invention provides a feed forward, seg- 
mented neural network having multiple network lay- 
ers and a plurality of subnetworks. Each subnet- 

45 work resides within one of the multiple network 
layers and at least one of the network layers con- 
tains multiple subnetworks. All subnetworks within 
a network layer having multiple subnetworks are 
isolated or segmented from the other subnetworks 

so in the network layer. The multiple network layers 
are arranged in pyramidal fashion from an input 
network layer to an output network layer such that 
the number of subnetworks in the network layers 
decreases from the input network layer to the out- 

55 put network layer. In a hardware implementation, 
the subnetworks comprise memory devices. Fur- 
ther enhancements of the novel segmented neural 
network are also described. 
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In another aspect, a method for fabricating a 
high speed, feed forward, segmented neural net- 
work is set forth. The method includes the steps of: 
selecting a subnetwork model and a segmented 
neural network model employing the selected sub- 
network model (such that the segmented neural 
network model has multiple network layers within 
which multiple subnetworks are disposed); training 
the selected segmented neural network while im- 
posing a limitation that values passing from a net- 
work layer to an adjacent network layer comprise 
binary signals; 

mapping input and output values of each sub- 
network in the segmented neural network model; 
and 

programming mapped subnetwork in- 
puts/outputs into a memory device. 

Advantageous Effect of the Invention 

The neural network design presented herein 
allows processing speeds which are unmatched in 
todays neural network state-of-the-art technology, 
while at the same time maintaining simplicity of 
design. This is achieved by not fully interconnec- 
ting the network, termed herein "segmentation" of 
the network. In addition, by not fully interconnec- 
ting the network, the design can be scaled up to 
large input neural networks which would be virtually 
impossible to implement in hardware as a fully 
connected neural network. The speed of the neural 
network is dependent only on the number of net- 
work layers and the speed of the memory devices 
employed. 

Brief Description of the Drawings 

These and other objects, advantages and fea- 
tures of the present invention will be more readily 
understood from the following detailed description 
of certain preferred embodiments thereof, when 
considered in conjunction with the accompanying 
drawings in which: 

Fig. 1 is a representation of one embodiment of 
a segmented neural network pursuant to the 
present invention; 

Fig. 2 is an internal representation of one em- 
bodiment of a subnetwork pursuant to the 
present invention for use in the segmented neu- 
ral network of Fig. 1; 

Fig. 3 is a representation of an individual node 
in the subnetwork of Fig. 2; 
Fig. 4 is an example of a PROM subnetwork 
implementation pursuant to the present invention 
for the neural network of Fig. 1; and 
Fig. 5 is a flowchart of a neural network fabrica- 
tion technique pursuant to the present invention. 
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Detailed Description of the Invention 

In accordance with the present invention, a 
neural network system is provided which lends 

5 itself in implementation to a realizable amount of 
hardware and which is capable of high speed pat- 
tern recognition, with a large number of inputs and 
nodes. The system is referred to herein as a "seg- 
mented" neural network system and includes a 

w plurality of network layers stacked in an ascending 
pyramid fashion. The network layers are structured 
with a plurality of "subnetworks". Within each sub- 
network exists a plurality of nodes structured in a 
fully interconnected and/or partially interconnected 

75 layered arrangement. Each subnetwork contains a 
plurality of inputs and, in the embodiment de- 
scribed below, has one half the number of outputs 
compared to inputs. The inputs and outputs of the 
subnetwork are one bit digital values constrained to 

20 0 and 1 . Any number of nodes with any number of 
layers may be modeled within each subnetwork. 
Further, the number of subnetwork outputs could 
be other than one half the number of inputs. 

The first network layer is the input network 

25 layer. The last network layer is the output network 
layer. All other network layers in between are hid- 
den network layers. Each network layer can be 
modeled with any number of subnetworks, wherein 
each subnetwork is independent of all other sub- 

30 networks in the subject network layer. In this way, 
each network layer is segmented. 

For N inputs per subnetwork and M subnet- 
works in a given network layer, the network layer 
has NxM inputs and, in the preferred embodiment 

35 described herein, (NxM)/2 outputs. The outputs 
from each network layer go to the inputs of the 
next network layer such that the outputs from 
preselected subnetworks in the previous network 
layer connect to one subnetwork in the next layer. 

40 The size of each subsequent network layer is dic- 
tated by its lower network layer, i.e., the number of 
inputs needed for the next network layer in the 
pyramid. The number of inputs, subnetworks, and 
outputs in each subsequent network layer is re- 

45 duced from (e.g., one half) the previous layer. 
Network layers are stacked onto previous network 
layers until (for example) only one subnetwork is 
left in the top most network layer. In this way a 
pyramidal, segmented neural network is created. 

50 By creating subnetworks with finite and tech- 

nologically feasible numbers of inputs and outputs, 
the subnetworks can be replaced by simple mem- 
ory devices such as RAM or PROM look-up tables. 
After the network has been trained on a computer 

55 simulation, the input and output values of each 
subnetwork can be mapped and stored into a 
memory device. Then each subnetwork can be 
replaced with its corresponding memory device. 
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Once the subnetworks are replaced with hardware 
memory devices they can be used to accurately 
emulate the neural network in forward mode. The 
speed of the system will be high and largely dic- 
tated by the access time of the memory devices 
used, and the number of network layers in the 
structure. In addition, the number of physical inter- 
connections will be relatively low. 

A more detailed implementation of the inven- 
tion is next described with reference to the draw- 
ings wherein the same reference numbers are used 
through different figures to designate the same or 
similar components. 

One embodiment of a neural network system, 
generally denoted 10, pursuant to the present in- 
vention is depicted in Fig. 1. Neural network sys- 
tem 10 is a pyramidal, segmented neural network 
having five network layers labeled "multilayer 1" 
(i.e., the input layer), "multilayer 2", "multilayer 3", 
"multilayer 4" and "multilayer 5" (i.e., the output 
layer). With the exception of "multilayer 5", each 
layer of neural network 10 contains multiple sub- 
networks "SN" 12, with each "subnetwork" as- 
sumed to consist of 16 inputs and 8 outputs. Each 
subnetwork comprises a memory device such as a 
PROM, with the number of outputs from each sub- 
network being one half the number of inputs. For 
example, the outputs from two paired subnetworks 
in "multilayer 1" feed the same subnetwork in 
"multilayer 2", while the outputs from two paired 
subnetworks in "multilayer 2" feed the same sub- 
network in "multilayer 3", the outputs from two 
paired subnetworks in "multilayer 3" feed the same 
subnetwork input in "multilayer 4", and the outputs 
from the two subnetworks 12 in "multilayer 4" are 
fed as input to the single subnetwork comprising 
"multilayer 5". (As described below, in system 10 
the paring of subnetwork 12 outputs is predefin- 
ed.) 

Network 10 is used in forward mode only, with 
training taking place on a separate computer, and 
the results of that training being programmed into 
the subnetworks, or more particularly, the memory 
devices employed. Thus, in implementation the 
structure of the overall network is one consisting of 
multiple subnetworks, with each subnetwork, for 
example, consisting of 16 inputs and 8 outputs. 
The 8 outputs from each of the first layer sub- 
networks are combined 16 bits at a time as inputs 
to the second layer subnetworks. This structure is 
continued to the final layer where the 8 outputs 
comprise the results of the overall network. 

Neural network system 10 is resident within a 
computer processing system, such as a pattern 
recognition system, and receives input information 
to "multilayer 1" by way of a plurality of input 
latches 14 which are coupled to receive data di- 
rectly from the computer data bus 16. By way of 



example, 32 8-bit input latches (type 74ALS273) 
may be employed in a 256 input neural network. 
Information from prepaired latches is asynchro- 
nously fed to a corresponding subnetwork in "mul- 

5 tilayer 1". Input latches 14 are synchronously load- 
ed via the common data bus by activating the 
appropriate enable line EN[0...31]. Each subnet- 
work in the pyramidal neural network structure may 
be implemented as an AM27512 64k PROM which, 

to again, has 16 inputs and 8 outputs. Output from the 
neural network (i.e., from "multilayer 5") is buffered 
in an output buffer 18 for ultimate placement onto 
computer data bus 16. Buffer 18 may comprise 
integrated circuit type 74ALS244. 

75 By way of further example, one embodiment of 
subnetwork 12 pursuant to the present invention is 
shown in Fig. 2. In this embodiment, three nodal 
layers are depicted. An input nodal layer 20 com- 
prises a plurality of nodes 22, each of which re- 

20 ceives the 16 inputs to subnetwork 12. Output from 
input nodal layer 20 is provided to an intermediate 
nodal layer 24, the output of which is input to 
output nodal layer 26. The output from nodal layer 
26 comprises the output of subnetwork 12. This 

25 embodiment of a subnetwork comprises a fully 
interconnected subnetwork since each node in 
each layer receives the same inputs and each 
node outputs to each node in the subsequent nodal 
layer. 

30 As depicted in Fig. 3, each internal node 22 
comprises a non-linear summing agent. The node 
sums its inputs (x) multiplied by their respective 
weights (w1 , w2...wn) and then applies the sum to 
the non-linear transformation. The non-linear trans- 

35 formation has the characteristic of being bounded 
asymptotically in both the negative and positive 
infinity directions, and also has the characteristic of 
being monotonically increasing. There are several 
common functions which have this characteristic, 

40 one of which is the Tanh(x) function. Those skilled 
in the art will recognize that other functions may 
also meet these characteristics. 

As noted initially with respect to Fig. 1, the 
network is made up of layers of memory devices 

45 wherein each memory device has 16 inputs and 8 
outputs. By way of example, Fig. 4 depicts pin 
connections for an AM27512 type PROM device. 
Again, other memory devices could be employed. 
Each memory device is thought of as a subnetwork 

50 which (in the embodiment described herein) as 16 
inputs and 8 output nodes as shown in Fig. 2. 

Fabrication of a neural network pursuant to the 
present invention is next described with reference 
to Fig. 5. During training of the network, the net- 

55 work is modeled as a segmented network where 
the inputs are taken, for example, 16 at a time and 
presented to the corresponding individual subnet- 
works, 30 "Model Feed Forward Network As Seg- 
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mented Network." The subnetworks are modeled 
as 16 input fully interconnected multilayer networks 
with 8 output nodes. Any commercially available 
computer simulation technique may be used to 
train the network, however, certain limitations are 
applied, 32 "Train Network Using Computer Sim- 
ulation and Applying Limitations." Specifically, 
training is conducted such that the values which 
are passed from one network layer to the next are 
binary, i.e., "0" or "1 ". Within the subnetwork this 
condition does not apply and during training the 
internal states of the subnetwork are allowed to 
take on continuous real values. 

Once network training is completed, a com- 
plete input/output mapping is done on each sub- 
network, 34 "Conduct Complete Input/Output Map- 
ping for Each Subnetwork." With a complete in- 
put/output mapping of the subnetwork, internal 
states are no longer needed and the input/output 
relationship can be programmed directly into the 
selected memory device, 36 "Program the In- 
put/Output Relationships Into Memory Devices." 

For the network presented herein, there are 
256 binary inputs (see Fig. 1). The binary inputs 
are latched one byte at a time from a computer 
interface. These 256 inputs are then presented to 
the 16 input layer subnetwork memory devices, 
labeled "multilayer 1" in Fig. 1. Each 16 input 
subnetwork memory devices generates 8 outputs 
to be presented to the next network layer. The 
outputs are presented asynchronously as inputs to 
the next network layer. Thus, each subsequent 
network layer contains one half the number of 
memory devices as the previous layer, so that the 
second layer, labeled "multilayer 2" in Fig. 1, must 
have 8 memory devices and so on until the last 
layer of the pyramidal structure, i.e., "multilayer 5", 
has one memory device. The output layer has 8 
outputs which allows the network to make up to 
256 decisions or class distinctions. 

Once all 256 inputs are latched into the input of 
the network, the output of the network is delayed 
only by the propagation delay of the memory de- 
vices from one layer to the next. For example, for 
the memory device depicted in Fig. 4, the in- 
put/output delay is 30 nanoseconds. Thus, for 5 
layers, the entire network input/output delay is ap- 
proximately 1 50 nanoseconds. 

Since each subnetwork can reasonably be 
thought of as a 16 input, 3 layer, 8 output, fully 
interconnected network with 1 1 hidden layer nodes, 
each subnetwork model contains 266 intercon- 
nects. The overall network consists of 31 sub- 
networks so that the interconnections between sub- 
networks add up to 240. The total number of equiv- 
alent interconnects is thus 31 x 264 + 240 = 
8,184. This corresponds to an amazing 54 billion 
interconnects per second, which is at least an order 
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of magnitude greater than speeds attainable by any 
pre-existing neural network structure. 

Although specific embodiments of the present 
invention have been illustrated in the accompany- 

5 ing drawings and described in the foregoing de- 
tailed description, it will be understood that the 
invention is not limited to the particular embodi- 
ments described herein, but is capable of numer- 
ous rearrangements, modifications and substitu- 

10 tions without departing from the scope of the inven- 
tion. The following claims are intended to encom- 
pass all such modifications. 

Claims 

75 

1. A feed forward, segmented neural network (10) 
of the type including multiple network layers 
(12) and a plurality of subnetworks, character- 
ized by each subnetwork residing within one of 

20 said multiple network layers, at least one of 
said multiple network layers having multiple 
subnetworks and all of said network layers 
having at least one subnetwork, all subnet- 
works within each said network layers having 

25 multiple subnetworks (20, 22, 24, 26) being 

segmented from other subnetworks within said 
network layer, said multiple network layers be- 
ing arranged in pyramidal fashion from an in- 
put network layer (14) to an output network 

30 layer (26) such that the number of subnetworks 

in said multiple network layers decreases from 
said input network layer to said output network 
layer. 

35 2. The feed forward, segmented neural network 
of claim 1, wherein each of said plurality of 
subnetworks is implemented in hardware as a 
memory device. 

40 3. The feed forward, segmented neural network 
of claim 2, wherein each of said memory de- 
vices comprise either a RAM device or a 
PROM device. 

45 4. The feed forward, segmented neural network 
of claim 1, wherein each said memory device 
has M inputs and N outputs, wherein M>N. 

5. The feed forward, segmented neural network 
so of claim 4, wherein each said memory device 

has one half the number of outputs N com- 
pared to inputs M, N = M/2, such that each 
neural network layer is one-half the size of a 
previous network layer. 

55 

6. The feed forward, segmented neural network 
of claim 5, wherein each said memory device 
has 16 inputs and 8 outputs. 
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7. The feed forward, segmented neural network 
of claim 1 , wherein each said subnetwork com- 
prises one of a fully connected or a partially 
connected neural network. 

5 

& The feed forward, segmented neural network 
of claim 7, wherein each said subnetwork con- 
tains a plurality of sublayers, each of said 
sublayers having a plurality of processing 
nodes. io 



9. A method for fabricating a feed forward, seg- 
mented neural network, said method compris- 
ing the steps of: 

(a) selecting a subnetwork model and a ;s 
segmented neural network model employing 

the selected subnetwork model, said seg- 
mented neural network model having mul- 
tiple network layers within which multiple 
subnetworks are disposed; zo 

(b) training the segmented neural network 
selected in said step (a) while requiring that 
values passing from each network layer to 
an adjacent network layer comprise binary 
signals; 25 

(c) subsequent completion of said step (b), 
mapping input and output values of each 
subnetwork in said segmented neural net- 
work model; and 

(d) for each subnetwork, programming sub- 30 
network input/outputs mapped in said step 

(c) into a binary device. 

10. The fabricating method of claim 11, wherein 

said selecting step (a) further includes select- 35 
ing a segmented neural network model such 
that the total number of inputs, subnetworks, 
and outputs of each network layer of the model 
is one-half the number the total number of 
inputs, subnetworks, and outputs of an adja- 40 
cent network layer. 
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